Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

Convolutional neural networks have been widely deployed in various application scenarios. In order to extend the applications' boundaries to some accuracy-crucial domains, researchers have been investigating approaches to boost accuracy through either deeper or wider network structures, which brings with them the exponential increment of the computational and storage cost, delaying the responding time. In this paper, we propose a general training framework named self distillation, which notably enhances the performance (accuracy) of convolutional neural networks through shrinking the size of the network rather than aggrandizing it. Different from traditional knowledge distillation - a knowledge transformation methodology among networks, which forces student neural networks to approximate the softmax layer outputs of pre-trained teacher neural networks, the proposed self distillation framework distills knowledge within network itself. The networks are firstly divided into several sections. Then the knowledge in the deeper portion of the networks is squeezed into the shallow ones. Experiments further prove the generalization of the proposed self distillation framework: enhancement of accuracy at average level is 2.65%, varying from 0.61% in ResNeXt as minimum to 4.07% in VGG19 as maximum. In addition, it can also provide flexibility of depth-wise scalable inference on resource-limited edge devices.Our codes will be released on github soon.Comment: 10page

DeePMD-kit v2: A software package for Deep Potential models

DeePMD-kit is a powerful open-source software package that facilitates molecular dynamics simulations using machine learning potentials (MLP) known as Deep Potential (DP) models. This package, which was released in 2017, has been widely used in the fields of physics, chemistry, biology, and material science for studying atomistic systems. The current version of DeePMD-kit offers numerous advanced features such as DeepPot-SE, attention-based and hybrid descriptors, the ability to fit tensile properties, type embedding, model deviation, Deep Potential - Range Correction (DPRc), Deep Potential Long Range (DPLR), GPU support for customized operators, model compression, non-von Neumann molecular dynamics (NVNMD), and improved usability, including documentation, compiled binary packages, graphical user interfaces (GUI), and application programming interfaces (API). This article presents an overview of the current major version of the DeePMD-kit package, highlighting its features and technical details. Additionally, the article benchmarks the accuracy and efficiency of different models and discusses ongoing developments.Comment: 51 pages, 2 figure

Off-line evaluation of indoor positioning systems in different scenarios: the experiences from IPIN 2020 competition

Every year, for ten years now, the IPIN competition has aimed at evaluating real-world indoor localisation systems by testing them in a realistic environment, with realistic movement, using the EvAAL framework. The competition provided a unique overview of the state-of-the-art of systems, technologies, and methods for indoor positioning and navigation purposes. Through fair comparison of the performance achieved by each system, the competition was able to identify the most promising approaches and to pinpoint the most critical working conditions. In 2020, the competition included 5 diverse off-site off-site Tracks, each resembling real use cases and challenges for indoor positioning. The results in terms of participation and accuracy of the proposed systems have been encouraging. The best performing competitors obtained a third quartile of error of 1 m for the Smartphone Track and 0.5 m for the Foot-mounted IMU Track. While not running on physical systems, but only as algorithms, these results represent impressive achievements.Track 3 organizers were supported by the European Union’s Horizon 2020 Research and Innovation programme under the Marie Skłodowska Curie Grant 813278 (A-WEAR: A network for dynamic WEarable Applications with pRivacy constraints), MICROCEBUS (MICINN, ref. RTI2018-095168-B-C55, MCIU/AEI/FEDER UE), INSIGNIA (MICINN ref. PTQ2018-009981), and REPNIN+ (MICINN, ref. TEC2017-90808-REDT). We would like to thanks the UJI’s Library managers and employees for their support while collecting the required datasets for Track 3. Track 5 organizers were supported by JST-OPERA Program, Japan, under Grant JPMJOP1612. Track 7 organizers were supported by the Bavarian Ministry for Economic Affairs, Infrastructure, Transport and Technology through the Center for Analytics-Data-Applications (ADA-Center) within the framework of “BAYERN DIGITAL II. ” Team UMinho (Track 3) was supported by FCT—Fundação para a Ciência e Tecnologia within the R&D Units Project Scope under Grant UIDB/00319/2020, and the Ph.D. Fellowship under Grant PD/BD/137401/2018. Team YAI (Track 3) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 109-2221-E-197-026. Team Indora (Track 3) was supported in part by the Slovak Grant Agency, Ministry of Education and Academy of Science, Slovakia, under Grant 1/0177/21, and in part by the Slovak Research and Development Agency under Contract APVV-15-0091. Team TJU (Track 3) was supported in part by the National Natural Science Foundation of China under Grant 61771338 and in part by the Tianjin Research Funding under Grant 18ZXRHSY00190. Team Next-Newbie Reckoners (Track 3) were supported by the Singapore Government through the Industry Alignment Fund—Industry Collaboration Projects Grant. This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is a collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU). Team KawaguchiLab (Track 5) was supported by JSPS KAKENHI under Grant JP17H01762. Team WHU&AutoNavi (Track 6) was supported by the National Key Research and Development Program of China under Grant 2016YFB0502202. Team YAI (Tracks 6 and 7) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 110-2634-F-155-001

Off-Line Evaluation of Indoor Positioning Systems in Different Scenarios: The Experiences From IPIN 2020 Competition

Every year, for ten years now, the IPIN competition has aimed at evaluating real-world indoor localisation systems by testing them in a realistic environment, with realistic movement, using the EvAAL framework. The competition provided a unique overview of the state-of-the-art of systems, technologies, and methods for indoor positioning and navigation purposes. Through fair comparison of the performance achieved by each system, the competition was able to identify the most promising approaches and to pinpoint the most critical working conditions. In 2020, the competition included 5 diverse off-site off-site Tracks, each resembling real use cases and challenges for indoor positioning. The results in terms of participation and accuracy of the proposed systems have been encouraging. The best performing competitors obtained a third quartile of error of 1 m for the Smartphone Track and 0.5 m for the Foot-mounted IMU Track. While not running on physical systems, but only as algorithms, these results represent impressive achievements.Track 3 organizers were supported by the European Union’s Horizon 2020 Research and Innovation programme under the Marie Skłodowska Curie Grant 813278 (A-WEAR: A network for dynamic WEarable Applications with pRivacy constraints), MICROCEBUS (MICINN, ref. RTI2018-095168-B-C55, MCIU/AEI/FEDER UE), INSIGNIA (MICINN ref. PTQ2018-009981), and REPNIN+ (MICINN, ref. TEC2017-90808-REDT). We would like to thanks the UJI’s Library managers and employees for their support while collecting the required datasets for Track 3. Track 5 organizers were supported by JST-OPERA Program, Japan, under Grant JPMJOP1612. Track 7 organizers were supported by the Bavarian Ministry for Economic Affairs, Infrastructure, Transport and Technology through the Center for Analytics-Data-Applications (ADA-Center) within the framework of “BAYERN DIGITAL II. ” Team UMinho (Track 3) was supported by FCT—Fundação para a Ciência e Tecnologia within the R&D Units Project Scope under Grant UIDB/00319/2020, and the Ph.D. Fellowship under Grant PD/BD/137401/2018. Team YAI (Track 3) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 109-2221-E-197-026. Team Indora (Track 3) was supported in part by the Slovak Grant Agency, Ministry of Education and Academy of Science, Slovakia, under Grant 1/0177/21, and in part by the Slovak Research and Development Agency under Contract APVV-15-0091. Team TJU (Track 3) was supported in part by the National Natural Science Foundation of China under Grant 61771338 and in part by the Tianjin Research Funding under Grant 18ZXRHSY00190. Team Next-Newbie Reckoners (Track 3) were supported by the Singapore Government through the Industry Alignment Fund—Industry Collaboration Projects Grant. This research was conducted at Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU), which is a collaboration between Singapore Telecommunications Limited (Singtel) and Nanyang Technological University (NTU). Team KawaguchiLab (Track 5) was supported by JSPS KAKENHI under Grant JP17H01762. Team WHU&AutoNavi (Track 6) was supported by the National Key Research and Development Program of China under Grant 2016YFB0502202. Team YAI (Tracks 6 and 7) was supported by the Ministry of Science and Technology (MOST) of Taiwan under Grant MOST 110-2634-F-155-001.Peer reviewe

Towards Predicting the Measurement Noise Covariance with a Transformer and Residual Denoising Autoencoder for GNSS/INS Tightly-Coupled Integrated Navigation

The tightly coupled navigation system is commonly used in UAV products and land vehicles. It adopts the Kalman filter to combine raw satellite observations, including the pseudorange, pseudorange rate and Doppler frequency, with the inertial measurements to achieve high navigational accuracy in GNSS-challenged environments. The accurate estimation of measurement noise covariance can ensure the quick convergence of the Kalman filter and the accuracy of the navigation results. Existing tightly coupled integrated navigation systems employ either constant noise covariance or simple noise covariance updating methods, which cannot accurately reflect the dynamic measurement noises. In this article, we propose an adaptive measurement noise estimation algorithm using a transformer and residual denoising autoencoder (RDAE), which can dynamically estimate the covariance of measurement noise. The residual module is used to solve the gradient degradation problem. The DAE is adopted to learn the essential characteristics from the noisy ephemeris data. By introducing the attention mechanism, the transformer can effectively learn the time and space dependency of long-term ephemeris data, and thus dynamically adjusts the noise covariance with the predicted factors. Extensive experimental results demonstrate that our method can achieve sub-meter positioning accuracy in the outdoor open environment. In a GNSS-degraded environment, our proposed method can still obtain about 3 m positioning accuracy. Another test on a new dataset also confirms that our proposed method has reasonable robustness and adaptability

Cytosine methylation alteration in natural populations of Leymus chinensis induced by multiple abiotic stresses.

BACKGROUND: Human activity has a profound effect on the global environment and caused frequent occurrence of climatic fluctuations. To survive, plants need to adapt to the changing environmental conditions through altering their morphological and physiological traits. One known mechanism for phenotypic innovation to be achieved is environment-induced rapid yet inheritable epigenetic changes. Therefore, the use of molecular techniques to address the epigenetic mechanisms underpinning stress adaptation in plants is an important and challenging topic in biological research. In this study, we investigated the impact of warming, nitrogen (N) addition, and warming+nitrogen (N) addition stresses on the cytosine methylation status of Leymus chinensis Tzvel. at the population level by using the amplified fragment length polymorphism (AFLP), methylation-sensitive amplified polymorphism (MSAP) and retrotransposon based sequence-specific amplification polymorphism (SSAP) techniques. METHODOLOGY/PRINCIPAL FINDINGS: Our results showed that, although the percentages of cytosine methylation changes in SSAP are significantly higher than those in MSAP, all the treatment groups showed similar alteration patterns of hypermethylation and hypomethylation. It meant that the abiotic stresses have induced the alterations in cytosine methylation patterns, and the levels of cytosine methylation changes around the transposable element are higher than the other genomic regions. In addition, the identification and analysis of differentially methylated loci (DML) indicated that the abiotic stresses have also caused targeted methylation changes at specific loci and these DML might have contributed to the capability of plants in adaptation to the abiotic stresses. CONCLUSIONS/SIGNIFICANCE: Our results demonstrated that abiotic stresses related to global warming and nitrogen deposition readily evoke alterations of cytosine methylation, and which may provide a molecular basis for rapid adaptation by the affected plant populations to the changed environments

High Chromosomal Stability and Immortalized Totipotency Characterize Long-Term Tissue Cultures of Chinese Ginseng (Panax ginseng)

Chinese ginseng (Panax ginseng C. A. Meyer) is a highly cherished traditional Chinese medicine, with several confirmed medical effects and many more asserted health-boosting functions. Somatic chromosomal instability (CIN) is a hallmark of many types of human cancers and also related to other pathogenic conditions such as miscarriages and intellectual disabilities, hence, the study of this phenomenon is of wide scientific and translational medical significance. CIN also ubiquitously occurs in cultured plant cells, and is implicated as a major cause of the rapid decline/loss of totipotency with culture duration, which represents a major hindrance to the application of transgenic technologies in crop improvement. Here, we report two salient features of long-term cultured callus cells of ginseng, i.e., high chromosomal stability and virtually immortalized totipotency. Specifically, we document that our callus of ginseng, which has been subcultured for 12 consecutive years, remained highly stable at the chromosomal level and showed little decline in totipotency. We show that these remarkable features of cultured ginseng cells are likely relevant to the robust homeostasis of the transcriptional expression of specific genes (i.e., genes related to tissue totipotency and chromosomal stability) implicated in the manifestation of these two complex phenotypes. To our knowledge, these two properties of ginseng have not been observed in any animals (with respect to somatic chromosomal stability) and other plants. We posit that further exploration of the molecular mechanisms underlying these unique properties of ginseng, especially somatic chromosomal stability in protracted culture duration, may provide novel clues to the mechanistic understanding of the occurrence of CIN in human disease

Synergistic strain engineering of perovskite single crystals for highly stable and sensitive X-ray detectors with low-bias imaging and monitoring

X-ray detectors based on dual-site-doped perovskite single crystals exhibit excellent sensitivity of 2.6 x 10(4) mu C Gy(air)(-1) cm(-2) under a low field of 1 V cm(-1). The detectable dose rate is as low as 7.09 nGy(air) s(-1). The operational stability is beyond half a year. Although three-dimensional metal halide perovskite (ABX(3)) single crystals are promising next-generation materials for radiation detection, state-of-the-art perovskite X-ray detectors include methylammonium as A-site cations, limiting the operational stability. Previous efforts to improve the stability using formamidinium-caesium-alloyed A-site cations usually sacrifice the detection performance because of high trap densities. Here we successfully solve this trade-off between stability and detection performance by synergistic composition engineering, where we include A-site alloys to decrease the trap density and B-site dopants to release the microstrain induced by A-site alloying. As such, we develop high-performance perovskite X-ray detectors with excellent stability. Our X-ray detectors exhibit high sensitivity of (2.6 +/- 0.1) x 10(4) mu C Gy(air)(-1) cm(-2) under 1 V cm(-1) and ultralow limit of detection of 7.09 nGy(air) s(-1). In addition, they feature long-term operational stability over half a year and impressive thermal stability up to 125 degrees C. We further demonstrate the promise of our perovskite X-ray detectors for low-bias portable applications with high-quality X-ray imaging and monitoring prototypes.Funding Agencies|National Natural Science Foundation of China [62004182, 61875072, 51863013, 61874052]; Key Research and Development Projects of Jilin Science and Technology Department [20200401044GX]; Excellent Young Foundation of Jiangxi Province [20192BCB23009]; Knut and Alice Wallenberg Foundation [KAW 2019.0082]; Swedish Government Strategic Research Area in Materials Science on Functional Materials at Linkoping University ((faculty grant SFO-Mat-LiU) [2009-00971]</p

Examples of SSAP analysis on the control and treatment groups.

<p>Typical variations are marked by arrowheads. (Primer: BARE-1/H/M+TAG).</p